-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stop sync on a null value in a cursor column #19889
Stop sync on a null value in a cursor column #19889
Conversation
/test connector=connectors/source-postgres
Build FailedTest summary info:
|
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|---|---|---|
source-alloydb |
1.0.17 |
✅ | ✅ |
source-alloydb-strict-encrypt |
1.0.17 |
✅ | 🔵 (ignored) |
source-bigquery |
0.2.3 |
✅ | ✅ |
source-clickhouse |
0.1.14 |
✅ | ✅ |
source-clickhouse-strict-encrypt |
0.1.14 |
✅ | 🔵 (ignored) |
source-cockroachdb |
0.1.18 |
✅ | ✅ |
source-cockroachdb-strict-encrypt |
0.1.18 |
✅ | 🔵 (ignored) |
source-db2 |
0.1.16 |
✅ | ✅ |
source-db2-strict-encrypt |
0.1.16 |
✅ | 🔵 (ignored) |
source-dynamodb |
0.1.0 |
✅ | ✅ |
source-jdbc |
0.3.5 |
⚠ (doc not found) |
⚠ (not in seed) |
source-mongodb-strict-encrypt |
0.1.19 |
⚠ (doc not found) |
🔵 (ignored) |
source-mongodb-v2 |
0.1.19 |
✅ | ✅ |
source-mssql |
0.4.25 |
✅ | ✅ |
source-mssql-strict-encrypt |
0.4.25 |
✅ | 🔵 (ignored) |
source-mysql |
1.0.15 |
✅ | ✅ |
source-mysql-strict-encrypt |
1.0.15 |
✅ | 🔵 (ignored) |
source-oracle |
0.3.21 |
✅ | ✅ |
source-oracle-strict-encrypt |
0.3.21 |
✅ | 🔵 (ignored) |
source-postgres |
1.0.31 |
✅ | ✅ |
source-postgres-strict-encrypt |
1.0.31 |
✅ | 🔵 (ignored) |
source-redshift |
0.3.15 |
✅ | ✅ |
source-scaffold-java-jdbc |
0.1.0 |
⚠ (doc not found) |
⚠ (not in seed) |
source-snowflake |
0.1.26 |
✅ | ✅ |
source-tidb |
0.2.1 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (0)
Connector | Version | Changelog | Publish |
---|
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml ), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
/test connector=connectors/source-postgres
Build FailedTest summary info:
|
/test connector=connectors/source-postgres
Build FailedTest summary info:
|
/test connector=connectors/source-postgres
Build FailedTest summary info:
|
/test connector=connectors/source-postgres
|
/test connector=connectors/source-postgres
Build PassedTest summary info:
|
/test connector=connectors/source-postgres
|
/test connector=connectors/source-postgres
Build FailedTest summary info:
|
/test connector=connectors/source-postgres
|
/test connector=connectors/source-postgres
Build PassedTest summary info:
|
/test connector=connectors/source-postgres
Build PassedTest summary info:
|
/publish connector=connectors/source-postgres
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/source-postgres-strict-encrypt auto-bump-version=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
What
This change will cause a sync to stop if one of the streams contains a column designated as a cursor that contains a null value.
How
This is adding another step to the verification at the beginning of a
read
in source postgres.We look for each stream's cursor column and if that column is NULLABLE, we look for a null value .
Because the lookup can be expensive on a large table, we are only going to do that if the column allows null values.
Otherwise it will immediately pass this test.
Recommended reading order
PostgresSource.java
for query implementationAbstractDbSource.java
for how the new logic is added to the read process